Towards Efficient Deep Inference for Mobile Applications

Mobile applications are bene€ting signi€cantly from the advancement in deep learning, e.g. providing new features. Given a trained deep learning model, applications usually need to perform a series of matrix operations based on the input data, in order to infer possible output values. Because of model computation complexity and increased model sizes, those trained models are usually hosted in the cloud. When mobile apps need to utilize those models, they will have to send input data over the network. While cloud-based deep learning can provide reasonable response time for mobile apps, it also restricts the use case scenarios, e.g. mobile apps need to have access to network. With mobile speci€c deep learning optimizations, it is now possible to employ device-based inference. However, because mobile hardware, e.g. GPU and memory size, can be very di‚erent and limited when compared to desktop counterpart, it is important to understand the feasibility of this new device-based deep learning inference architecture. In this paper, we empirically evaluate the inference eciency of three Convolutional Neural Networks using a benchmark Android application we developed. Based on our application-driven analysis, we have identi€ed several performance boŠlenecks for mobile applications powered by on-device deep learning inference. ACM Reference format: Tian Guo. 2017. Towards Ecient Deep Inference for Mobile Applications. In Proceedings of arxiv, 2017, July 2017 (arxiv’17), 7 pages. DOI: 10.1145/nnnnnnn.nnnnnnn